Lecture Notes on Information Theory Volume I by Po

Preface The reliable transmission of information bearing signals over a noisy communication channel is at the heart of what we call communication. Information theory—founded by Claude E. Shannon in 1948—provides a mathematical framework for the theory of communication; it describes the fundamental limits to how efficiently one can encode information and still be able to recover it with negligible loss. This course will examine the basic concepts of this theory. What follows is a tentative list of topics to be covered. (b) Fundamentals of channel coding: Discrete memoryless channels, mutual information, channel capacity, coding theorem for discrete mem-oryless channels, weak converse, channel capacity with output feedback , the Shannon joint source-channel coding theorem. (c) Source coding with distortion (rate distortion theory): Discrete mem-oryless sources, rate-distortion function and its properties, rate-distortion theorem. (d) Other topics: Information measures for continuous random variables, capacity of discrete-time and band-limited continuous-time Gaussian channels, rate-distortion function of the memoryless Gaussian source, encoding of discrete sources with memory, capacity of discrete channels with memory. (e) Fundamental backgrounds on real analysis and probability (Appendix): The concept of set, supremum and maximum, infimum and minimum, boundedness, sequences and their limits, equivalence, probability space, random variable and random process, relation between a source and a random process, convergence of sequences of random variables, ergodicity and laws of large numbers, central limit theorem, concavity and convexity, Jensen's inequality. ii 2. Volumn II: (a) General information measure: Information spectrum and Quantile and their properties, Rényi's informatino measures. (b) Advanced topics of losslesss data compression: Fixed-length lossless data compression theorem for arbitrary channels, Variable-length loss-less data compression theorem for arbitrary channels, entropy of En-glish, Lempel-Ziv code. (c) Measure of randomness and resolvability: Resolvability and source coding, approximation of output statistics for arbitrary channels. (d) Advanced topics of channel coding: Channel capacity for arbitrary single-user channel, optimistic Shannon coding theorem, strong capacity , ε-capacity. (e) Advanced topics of lossy data compressing (f) Hypothesis testing: Error exponent and divergence, large deviations theory, Berry-Esseen theorem. As shown from the list, the lecture notes are divided into two volumes. The first volume is suitable for a 12-week introductory course as that given at the It also meets the need of a fundamental course for senior undergraduates as that given at the the lecturer can selectively add advanced topics covered in the second volume to enrich the lecture content, and provide a more complete and advanced view on …